1. Field of the Invention
The present invention relates to image processing for correcting (or reducing) a deterioration component of an image generated by image pickup, utilizing an image restoration filter.
2. Description of the Related Art
An image obtained by capturing an object by an image pickup apparatus, such as a digital camera, contains a blur component as an image deterioration component, such as the spherical aberration, coma, curvature of field, and astigmatism of an image pickup optical system (simply referred to as an “optical system” hereinafter). This blur occurs when a light flux emitted from one point of the object, which is expected to converge on one point on an image pickup surface when there is no aberration or diffraction, forms an image with some divergence.
The blur, as used herein, is optically represented by a point spread function (“PSF”) and is different from a blur caused by defocusing. A color shift in a color image caused by longitudinal chromatic aberration, chromatic spherical aberration, or chromatic coma of the optical system is a difference of the blurring degree among wavelengths of the light. In addition, the lateral chromatic shift can be regarded as a position shift or a phase shift due to a difference of the image pickup magnification for each wavelength of the light if it is caused by the lateral chromatic aberration of the image pickup optical system.
An optical transfer function (“OTF”) obtained by a Fourier transform to the PSF is frequency component information of the aberration and is represented by a complex number. An absolute value of the OTF or an amplitude component is referred to as a modulation transfer function (“MTF”) and a phase component is referred to as a phase transfer function (“PTF”). The MTF and the PTF are frequency characteristics of an amplitude component and a phase component of the image deterioration due to the aberration. The phase component will be expressed as a phase angle by the following equation, where Re(OTF) and Im(OTF) are a real part and an imaginary part of the OTF:PTF=tan−1(Im(OTF)/Re(OTF))
Thus, the OTF of the image pickup optical system deteriorates the amplitude component and the phase component of the image, and the deteriorated image has an asymmetrical blur state of points of the object like the coma.
In addition, the lateral chromatic aberration (i.e., chromatic aberration of magnification) occurs when an imaging position shifts due to a difference of an imaging magnification for each wavelength of the light, and is obtained as RGB chromatic components, for example, in accordance with the spectral characteristic of the image pickup apparatus. The image divergence occurs not only when the image position shifts among the RGB, but also when the imaging position shifts for each wavelength of a chromatic component or when the phase shifts. Therefore, due to the phase degradation component of the aberration, asymmetry of the PSF is viewed on a primary section of each direction (azimuth direction) orthogonal to a principal ray (a ray that passes the center of the pupil of the image pickup optical system). The amplitude degradation component affects the size of the spread of the PSF for each azimuth direction.
In order to highly precisely correct through image processing the image degradation caused by the image pickup optical system, it is thus necessary to correct the phase degradation component and the amplitude degradation component of the aberration.
Edge enhancement processing is a known method of correcting the amplitude degradation component by detecting an edge portion in an image and by enhancing the edge as disclosed in Japanese Patent Laid-Open No. (“JP”) 2000-156816.
A geometric correction is a known method of correcting a phase degradation component by varying an image magnification for each chromatic component of an image as disclosed in JP 06-113309.
Another known method of correcting the amplitude degradation component and the phase degradation component utilizes the OTF information of the image pickup optical system. This method is referred to as an image restoration or image recovery, and processing used to correct (or reduce) an image degradation component utilizing the OTF information will be referred to as image restoration processing hereinafter.
The image restoration processing will be outlined as follows: Assume that g(x, y) is a deteriorated image (input image), f(x, y) is a non-deteriorated, original image, and h(x, y) is the PSF as a Fourier pair of the OTF. Then, the following equations are established, where * denotes a convolution and (x, y) denotes a coordinate on an image.g(x,y)=h(x,y)*f(x,y)
The above equation is converted into a display format on a two-dimensional frequency plane by a Fourier transform, which is a format of a product for each frequency. “H” denotes a Fourier-transformed PSF or corresponds to the OTF. (u, v) denotes a coordinate on the two-dimensional frequency plane or corresponds to a frequency.G(u,v)=H(u,v)·F(u,v)
In order to obtain the original image from the deteriorated image, both sides are divided by H as follows:G(u,v)/H(u,v)=F(u,v)
When this F(u, v) is inverse-Fourier-transformed and returned to the real surface, the restored image corresponding to the original image f(x, y) can be obtained.
Where R denotes an inverse Fourier transform of H−1, the original image can also be obtained through the convolution processing to the image on the real plane as in the following equation:g(x,y)*R(x,y)=f(x,y)
This R(x, y) is referred to as an image restoration filter. For a two-dimensional image, the image restoration filter is usually a two-dimensional filter having a tap (cell) corresponding to each pixel of the image. In general, as the number of taps in the image restoration filter increases, the restoration precision improves. The image restoration filter needs to reflect at least an aberrational characteristic, and is remarkably different from a conventional edge enhancement filter (high-pass filter) having about three taps in each of the horizontal and vertical directions. Since the image restoration filter is prepared based on the OTF, the deterioration of each of the amplitude component and the phase component can be highly precisely corrected.
The real image has a noise component, and the noise component is remarkably amplified disadvantageously along with the restoration of the deteriorated image when the restoration utilizes the image restoration filter made with a perfect reciprocal of the OTF. This is because the MTF (amplitude component) of the image pickup system is increased to 1 over the entire frequency range while the noise amplification is added to the amplitude component of the image.
The MTF corresponding to the amplitude degradation of the image pickup optical system is restored to 1, but the power spectrum of the noise is simultaneously increased and the noise is consequently amplified according to the MTF increasing degree (restoration gain). With noises, an image worth viewing cannot generally be obtained.
This can be expressed by the following equations where N denotes a noise component:G(u,v)=H(u,v)·F(u,v)+N(u,v)G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v)
For example, like a Wiener filter illustrated by Equation 1, there is a known method of controlling the restoration degree in accordance with a signal to noise ratio (“SNR”) between an image signal and a noise signal:
                              M          ⁡                      (                          u              ,              v                        )                          =                              1                          H              ⁡                              (                                  u                  ,                  v                                )                                              ⁢                                                                                      H                  ⁡                                      (                                          u                      ,                      v                                        )                                                                              2                                                                                                              H                    ⁡                                          (                                              u                        ,                        v                                            )                                                                                        2                            +                              SNR                2                                                                        Equation        ⁢                                  ⁢        1            
M(u, v) denotes a frequency characteristic of the Wiener filter, and |H(u, v)| denotes an absolute value (MTF) of the OTF. For each frequency, this method further restrains the restoration gain as the MTF is smaller, and further enhances the restoration gain as the MTF is larger. In general, the MTF of the image pickup optical system is high on the low frequency side and low on the high frequency side, and thus the restoration gain of the image is restrained substantially on the high frequency side.
When the noises are impermissible to the desired image quality after the restoration processing is performed by setting the actual SNR to a parameter, an additional restraint on the noise amplification is necessary by adjusting the SNR parameter so as to reduce the restoration gain. This corresponds to the SNR value in Equation 1 set to a larger value. In addition, when there is a difference between the aberrational characteristic in the actual image pickup condition and the aberrational characteristic estimated by the image restoration filter, the restored image may contain a problem of an artifact, such as ringing. Moreover, when the restoration degree is different from the estimated one for each of the RGB chromatic components, the restored image may be colored or a false color may occur. This false color is also one of the artifacts, such as noises and ringing.
FIG. 19 illustrates false colors in a restored image. The false color occurs as illustrated by a broken line in FIG. 19. A false color “A” occurs when an edge part is further colored than a pre-restoration image caused by the manufacturing error. A false color “B” occurs when a periphery of a brightness saturation part is further colored than the pre-restoration image caused by the brightness saturation generated by the intensity of solar reflected light. A false color “C” occurs when the edge part is further colored than the pre-restoration image caused by defocusing.
In the Wiener filter, the restoration degree lowers by setting a large value to the SNR value in Equation 1. The restoration degree starts dropping from a high frequency and the pre-restoration frequency characteristic cannot be reproduced even when the restoration degree is reduced. This is clear from the fact that M=1 is not established or the image restoration filter does not affect the image unless the SNR has a frequency characteristic in the Equation 1.
In order to adjust the restoration degree, JP 2007-183842 sets a parameter used to design the image restoration filter as in Equation 2:
                              F          ⁡                      (                          u              ,              v                        )                          =                                                            α                ⁢                                                                  ⁢                                                      H                    ⁡                                          (                                              u                        ,                        v                                            )                                                        *                                            +              1              -              α                                                      α                ⁢                                                                                                H                      ⁡                                              (                                                  u                          ,                          v                                                )                                                                                                  2                                            +              1              -              α                                ×                      G            ⁡                          (                              u                ,                v                            )                                                          Equation        ⁢                                  ⁢        2            
F(u, v) and G(u, v) are Fourier transforms of the restored image and the deteriorated image, respectively. The adjustment parameter a provides a change from a filter that affects nothing (α=0) to an inverse filter (α=1), and the restoration degree of the image is adjustable in a range from the original image to the maximum restored image.
As described above, the degradation of the image caused by the image pickup through the image pickup optical system contains the phase component and the amplitude component of the aberration, and thus it is necessary to correct the phase component, particularly, in order to correct the deletion of the image caused by the astigmatism.
The edge enhancement method disclosed in JP 2000-156816 does not utilize the aberrational characteristic, but enhances only the edge part, and can restrain the noises of the non-edge part. Nevertheless, this method cannot correct the phase component (PTF) of the aberration or the asymmetry, such as the coma and the lateral chromatic aberration. The edge enhancement cannot correct the deletion of the image in the periphery of the image.
The lateral chromatic aberration correcting method disclosed in JP 06-113309 can correct the color shift caused by the positional shifts among the chromatic components of the PSF, but cannot correct the asymmetry of the PSF itself for each chromatic component since this method is directed to the shift conversion of the image position. The PSF having the coma is asymmetric in the meridional direction, and thus the edge blur degrees are different between the side close to the optical axis (or the center of an image) and the side far from the optical axis. Moreover, due to the chromatic coma, the edge blur degrees are different between the side close to the optical axis and the side far from the optical axis for each chromatic component. Thus, the correction that equalizes the image magnifications among the chromatic components so as to make inconspicuous the color shift at the edge cannot optimize both the side close to the optical axis and the side far from the optical axis since the color shifts are different between both sides.
The image restoration method disclosed in JP 2007-183842 can correct the amplitude degradation component and the phase degradation component of the image but causes the noise amplification, as described above. The noise amplification occurs along with the correction of the amplitude component among the amplitude component and the phase component of the image. The image restoration filter disclosed in JP 2007-183842 is configured to lower a correction amount of the phase component as a correction amount of the amplitude component is lowered, and cannot correct only the phase component while maintaining the pre-restoration amplitude characteristic similar to the Wiener filter.